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Abstract 


IDRC has telecentre evaluation initiatives in Africa, Asia and Latin America. Each of these is in 
the process of establishing evaluation frameworks and each is adopting very different 
approaches. The current paper seeks to build on the outputs of IDRC’s Far Hills Workshop on 
Telecentre Evaluation (September 1999) to explore the experiences in the two regions. These 
explorations will in turn contribute to IDRC’s current initiative to develop a framework for ICTs 
evaluation. 


The Far Hills Workshop, and the resulting report entitled Telecentre Evaluation: A Global 
Perspective, had as objectives to explore challenges and opportunities for telecentre evaluation; 
understand and compare emerging evaluation frameworks and methodologies; assess needs and 
resources available; identify salient issues affecting telecentre performance; and provide an 
opportunity to exchange experiences and lessons across regions. 


Understanding Telecentres 


Telecentres are a new trend in international development. Loosely defined, telecentres are a physical 
space that provides public access to information and communication technologies, notably the Internet, 
for educational, personal, social, and economic development (for a more detailed discussion on 
telecentre definition see Gomez, 1999). When it comes to evaluating what kind of benefit or 
development telecentres bring to a community, we are only beginning to scrape the surface. About six 
months ago IDRC’s PAN and Acacia Networking initiatives organised an international meeting in 
Quebec, Canada, to discuss telecentre evaluation methodologies. At the meeting we established a list of 
criteria for telecentre evaluation (Gómez, 1999). Based on this work, we suggest the following 
guidelines to assess ICT evaluation that is useful, financially responsible, builds local capacity, and 
enables shared learning: 


Guiding Principles for sound Telecentre Evaluation 
Building from discussions at Global Telecentre Evaluation Meeting, Far Hills Inn, September 1999 
Guiding principles to help evaluation be USEFUL, FINANCIALLY RESPONSIBLE, BUILDS 


1 A draft of this paper was presented at the 50" Annual International Communication Association ICA Conference 
June 1-5, Acapulco, Mexico 
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LOCAL CAPACITY and ENABLES SHARED LEARNING: 


1) Participatory All relevant stakeholders, including users and non-users, are 
involved in the evaluation process. 


Socially Inclusive Evaluations explicitly address, include and provide 
differentiated information about the experiences of sub- groups 
in a society (gender, age, culture, religion, etc.). 


Locally Grounded Evaluations are context sensitive. (By way of explanation, this 
may imply relying on available expertise or taking into 
consideration local practices, world views and priorities.) 


Public and Transparent Evaluation results are publicized in ways appropriate to all 
relevant stakeholders. 
Evaluation processes are transparent. 


Methodologically Choices of methods and tools are appropriate for the context 
Appropriate and use, balancing replicability with usefulness and 
methodological robustness with practicality. 


6) Sustainability Enhancing Evaluations contribute to making telecentres more viable, to 
enhancing services and to making them more relevant. 


7) Capacity Building Lessons learned — both failures and successes - are 
documented during evaluations and are used to empower 
telecentre practitioners and users. Training is carried out where 
a need is identified. 


8) Reflective of Shared Evaluations are based on a common understanding of the 
Visions telecentre mission, the evaluation process, the goal of the 
evaluation, and how results will be used. 


9) Strategically Oriented The evaluation strategy is incorporated into the project design 
and reflects project objectives. It is focused on clear questions 
that are important to stakeholders. 


10) Gender Sensitive The evaluation strategy, process, methodology and tools are 
sensitive to the particular realities and needs of women. Women 
are consulted in the development and realization of evaluative 
processes. 


Gomez, Ricardo and Katherine Reilly. Comparing Approaches: Telecentre Evaluation Experiences in Asia, Africa 
and Latin America. (mimeograph) July 2000. 
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At the Far Hills meeting, several participants noted a distinction between two classes of evaluation 
criteria. The first is input measures, which Harris explains, "are not immediately associated with the 
benefits that the user community enjoy, but they are regarded as essential preconditions for those 
benefits to emerge. Consequently, they can be used by operators as guides for establishing telecentres 
and for constructing the services that they will provide" (1999). Output measures on the other hand, 
"relate to the achievements of the Telecentre within the community it serves. Output measures are 
distinguished from input measures in that they relate to the additional benefits that the community enjoys 
as a result of utilising the services of the centre" (Harris, 1999). Hudson makes a distinction between 
formative and summative functions of evaluation, the former being "feedback on the project or activity: 
how well is it working, what changes or improvements should be made, what was learned that could be 
applied in other similar projects, etc." and the later being, "Did the project achieve its goals? What was 
learned about how ICTs can contribute to social and economic development?" (Hudson, 1999). 
Formative evaluation focuses more on process and feedback, while summative evaluation seeks to 
discover if a difference was made by a project. 


These distinctions can be somewhat artificial, however. For example, how well a project is working is 
inextricably linked to whether it achieves its goals. A broader basis for understanding is required to 
compare and examine the evaluation work being done by IDRC. This is in part the intent of the Guiding 
Principles. But on another dimension, it is important to understand what the purpose of any evaluation is, 
and for an organisation like IDRC, which works on various levels, this is key. In the case of an 
international donor agency, evaluations can serve different purposes. 


Within IDRC, telecentre projects are often carried out in association with a recipient (often a regional 
body), which in turn works with various telecentre organisations. Many projects follow a three-tiered 
cascading structure (IDRC - recipient - telecentres), while in other cases IDRC program officers 
interact directly with telecentres. But, while IDRC’s research is often concerned with broader policy 
issues and global impacts— what could be termed the “big questions” — the immediate information needs 
of telecentre users and operators at the local level are more specialised.” In an environment of limited 
resources, the potential exists that choices will have to be made between research that answers the big 
questions and research that suits local information needs. Or, to be less absolute, it may be difficult to 
fully reconcile the need for data that is relevant for regional studies with the need for data that is relevant 
locally. Thus, there is a possibility that studies executed by IDRC or a regional organisation with the 
assistance — and even input — of local groups, will not fully meet the needs of those local groups. 
Alternatively, it is possible that specialised research at the local level will not contribute fully or 
meaningfully to larger research questions. 


Research activities which seek to answer the “big questions” are important for justifying international 
contributions to telecentre activities, determining regional caveats, understanding both positive and 
negative outcomes, discovering linkages with other development objectives, uncovering successful 
practices, and building capacity, among other things. The research will determine whether and how 
telecentre activities are carried out in the future, and, as such, it is important that these research activities 
have input from local groups. Evaluation cannot be viewed by primary and secondary recipients as 


2 This is not to say that local organizations do not get involved in the larger questions. In fact, much of IDRC’s 
work in the are of connectivity has focused on helping local groups contribute to regional or international 
exchanges. 
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simply a condition of funding. It should be viewed as a means to influence the activities of development 
agencies. Evaluations are an opportunity to determine that "This is working for us, but that is not." 


On the other hand, a goal of IDRC’s work is also to enhance the sustainability of individual telecentres. 
Telecentres are a relatively new phenomenon and there exists little guidance for the individual telecentre 
operator on how to successfully carry out a project. Thus, evaluation is particularly important at the local 
level so that the telecentre operator can better understand local needs and opportunities, so that 
problem areas can be identified and addressed, and so that successful or unsuccessful practices can be 
recorded. In order for telecentre operators to realise these local evaluations, they will require training 
and support. It is not clear that this area is being systematically addressed - either by IDRC or by other 
institutions. The lack of clarity may emerge from the fact that local evaluations are folded into larger 
comparative or regional evaluations. Thought information from such studies may be useful to local 
operators, it is not sufficient. Information needs at the local level are likely more frequent, detailed and 
specialised then larger studies can provide. Thus, it may be necessary to develop appropriate guidelines 
and training for evaluation at the local telecentre level. This may be difficult given the variety of telecentre 
experiences and communities served. However, it is possible that much knowledge already exists in the 
form of business auditing techniques and management reviews. Furthermore, telecentres can provide 
each other with a great deal of information and support if the appropriate channels are put in place for 
communication. 


When various levels of partnership emerge, evaluation becomes more complicated. IDRC is working on 
different approaches to telecentre evaluation in each of Latin America, Asia and Africa. The activities in 
each region are vastly different and each approach has its strengths and weaknesses. Throughout this 
paper, these approaches will be described and presented in light of the guiding principles to telecentre 
evaluation. The following chart synthesises the findings of this analysis and provides a means to compare 
approaches. The discussions also reflect the balance that each approach has struck between the 
individual needs of telecentres and the need to address larger research questions. The conclusion draws 
from these results to provide summary results, recommendations and future directions. 


Experiences in ASIA: PANTLEG - An emerging evaluation approach 


Evaluation of IDRC's Asian telecentre projects got its impetus during regional working sessions at the 
Far Hills workshop. At this time the PAN- Asia Telecentre Learning & Evaluation Group (PANTLEG) 
was established consisting of two IDRC staff, and five Asian partner organisations. The group decided 
to explore an innovative approach to evaluation. In order to measure the impacts of the telecentre 
initiatives, the group investigated the idea of collecting stories from telecentre operators and users. 
According to the PANLEG group: 


Stories, whilst anecdotal, offer a rich picture of the impact of ICT interventions in local, 
complex and dynamic social settings. They are accessible and verifiable during short visits 
and they acknowledge the often indirect influence that development interventions have on 
the behaviour of their beneficiaries. Moreover, stories as evaluation concede that the 
benefits of telecentre activities are often detectable only after they have been installed, 
contrary to traditional approaches to information systems, in which expected benefits are 
usually specified before the technology is installed. (PANTLEG, 1999) 
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Comparing Approaches to Telecentre Evaluation 


The Evaluation Should Be: | Asia: PANTLEG Latin America: TELELAC 

1) Participatory Great potential if local story gatherers are Guiding philosophy is participation. 
empowered. 

2) Socially Inclusive Individual voices can represent a variety of To be determined. 
groups. 


3) Locally Grounded Stories reflect local experiences. Adapted to the context of the telecentre user. 


4) Public and Transparent | Publication processes and level of Highly public and transparent processes and methods of 
transparency is unclear. dissemination. 


5) Methodologically Practical. Not particularly robust. Focus on Flexibility and adaptability are a challenge. 
Appropriate successes 


6) Sustainability Enhancing | Opportunity for learning and exchanges Goal is to enhance the sustainability of telecentres. 
7) Capacity Building Emphasis on celebrating success. Capacity-building and training. 


8) Reflective of Shared Unclear Strong collective process. 

Visions 

9) Strategically Oriented Open-ended. Methodology must be flexible to accommodate strategic 
goals. 


10) Gender Sensitive DEORE 
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Because stories must be told to an evaluator, and because a major criterion for evaluation as determined 
by the Far Hills workshop is participation, the group also explored the idea of having the telecentres 
evaluate each other. Thus, neighbouring telecentres would exchange, compare and document their 
experiences, providing an opportunity for both primary learning in the form of telecentre capacity 
building, and for secondary learning in the form of evaluation capacity building. The approach has been 
dubbed a ‘learning evaluation.’ 


A first exploration of this methodology has been carried out by PANTLEG. In November 1999, the 
group realised a mission to study the projects of two IDRC partner organisations in India. Two IDRC 
representatives accompanied Roger Harris of Malaysia, Merlita Opena of the Philippines and 
Narangerel Dander of Mongolia to visit V. Balaji of the MS Swaminathan Research Centre (MSSRF) 
and Loyola Joseph of the Foundation of Occupational Development (FOOD), both in Madras. This 
entire group of project co-ordinators visited six village telecentres, one of which acts as an information 
hub for three others administered by MSSRF, and one centre of operations hooked to one telecentre 
administered by FOOD. The group had discussions with telecentre users and operators to, "discover a 
range of typical stories that described instances of telecentre use which were perceived by the users to 
have led to successful aitcomes," and to, "discover any success stories concerning operational 
problems, and the means that were used to overcome them" (PANTLEG, 1999). Story collection 
typically started with village leaders (mostly male), then users (first male and then female users) and 
finally operators (in one case the operators were part of a woman's co-operative). In all, the group 
uncovered 24 success stories, each of which, "owes its outcome to the sensitive and timely delivery of 
useful information that contributed to local knowledge which facilitated something desirable for the 
recipient" (PANTLEG, 1999). 


The evaluation group summarised their thoughts for the 2 project organisers of FOOD and MSSRF. 
They then reviewed the impact assessment section of an MSSRF project proposal. MSSRF had 
proposed regular surveys of key indicators throughout their projects in order to measure improvements 
attributable to the telecentres. An important lesson of the exercise for the PANTLEG group was that 
evaluation of a telecentre project is not straightforward. They suggested that the proposal be changed so 
that baseline surveys were followed by story gathering and the collection of fresh empirical data, a 
process more akin to action research. This approach recognises that the implementation of the telecentre 
projects can shift the focus of the research question and generate new avenues for inquiry. Furthermore, 
the mix of approaches would provide a much richer picture than could come from a uniquely survey 

approach. Another recommendation was that MSSRF focus less on specific project outcomes which 
may be hard to quantify and measure. Instead, the evaluation group suggested that, "it might be 

beneficial to acknowledge ... uncertainty of outcomes at the outset and to demonstrate the capability of 
the project to tease out unpredictable benefits through the skilful and sensitive application of appropriate 
methods such as Action Research, Participatory Rural Appraisal, Rapid Rural Appraisal and Outcome 
Mapping" (PANTLEG, 1999). 


The mission to Asia gathered several stories indicating the positive impact that telecentres can have in 
underdeveloped communities. For example, in one instance, a farmer discovered late at night that if a 
cow didn’t receive treatment within a few hours, it would likely die. PANTLEG's report explains that, 
"The farmer was able to summon help from the telecentre staff who searched their networked 
information sources for a veterinary surgeon who would was close enough to apply treatment within a 
short time. The surgeon was contacted by telephone and he arrived in time to save the cow" 
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(PANTLEG, 1999). 


Story gathering has many positive characteristics. Itis highly focused on the individual, which provides a 
means to ensure social inclusion. Social groups and women have an opportunity to express their 
experiences in their own voice. This also makes the approach very locally grounded in that the stories 
will reflect the reality of the orators. As the PANTLEG group points out: 


The context of the evaluation supported the approach, by allowing individuals to express 
themselves freely and by focussing special attention during each session on the women 
beneficiaries. The Group had every reason to believe that the stories they heard were 
genuine and were faithfully transmitted through the translation process. However, we 
acknowledge that the evaluation, depending on its purpose, could in some ways be 
described as superficial. (PANTLEG, 1999) 


Indeed, the applicability of the approach may depend on the situation. By its nature, it is flexible and 
easily adaptable and stories are easily accessible and verifiable, however it is not a particularly robust 
methodology in terms of replicability and collection of baseline data for crosscutting comparisons. 
Furthermore, because the approach can uncover unanticipated impacts, it could be difficult to use for 
certain strategic purposes. An open-ended question is much more unwieldy than a set of survey 
questions, which can constrain the range of answers. Stories, although useful in some contexts, may not 
be considered sufficiently robust by government agencies, donors or partners. For some purposes 
(policy development, telecentre management, economic implications), a different type of evaluation may 
be more appropriate. 


PANTLEG has explored the possibility of using story gathering throughout a project to complement 
base-line empirical data gathered at the beginning of the project. Telecentres, as it was earlier pointed 
out, are implemented with a certain level of unpredictability: if you build it they will come - and benefit. 
The group asks, "is our ability to predict outcomes deficient in some way or is it more in the nature of 
the creativity and resourcefulness of individuals who have been empowered with access to information 
that we should not be expected to predict how they will conduct themselves with such new found 
capacities?" (PANTLEG, 1999). With telecentres the expected outcomes, as measured against project 
objectives or current knowledge, may not reflect the actual impacts - positive or negative. For program 
evaluation this is problematic as funding agents and policy makers may want a measurable impact. This 
suggests that learning evaluations might need to be supplemented with other kinds of data, including 
more quantitative information. In this case, story telling could be considered an indicator of areas that 
warrant more systematic research. 


The need to choose an appropriate methodology underlines the importance of involving telecentre users 
and operators in the planning and realisation of evaluations. The users and operators will have a position 
on what they consider to be a methodologically correct approach and a good strategic orientation. 
Furthermore, by providing an opportunity for the users and operators to work out these positions, the 
evaluation will become more reflective of a common vision, and the capacity of the telecentre community 
will be enhanced. During PANTLEG's visit to India, exchanges took place at the project level in a way 
that allowed program implementation teams to learn from each other. The mission also allowed the 
program implementers and IDRC staff members to collect information in a participatory manner by 
working with the telecentres. However, the telecentre operators did not have an opportunity to learn 
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from each other. It would have been interesting to foster evaluation through exchanges at the telecentre 
level. This would allow telecentre operators and users to exchange information on service provision and 
uses. Furthermore, it would make evaluations more participatory, strategic and reflective of the local 
vision of the telecentre community. 


Before these groups can be expected to carry out effective evaluations, training will be required. Story 
telling requires facilitation and interviewing skills. The evaluator would not simply listen and record 
stories, but would also need to effectively manage the information that is uncovered. In such an open- 
ended process, the stories told could unearth some difficult questions for the stakeholders. In this case, 
the evaluator may be called on to suggest solutions to the problem or moderate disputes. The approach 
also requires that evaluation results be used in an appropriate manner. Stories are sometimes personal, 
so there should be some discussion of the ethical aspects of how the information is used. Closely related 
to this is the imperative that the purposes and results of the study be made public in a manner 
appropriate to all stakeholders. 


Training on the approach would also help to overcome another potential pitfall. When two telecentres or 
two project co-ordination teams are brought together there is a risk that successes will be emphasised 
over failures, as was seen in the initial PANTLEG experiment. This particular project was operating 
successfully at the time of the evaluation, and as a result, the evaluation produced a list of positive 
outcomes. However, it should be noted that the language of the PANTLEG report on the mission to 
India (PANTLEG, 1999) does emphasise successes. Furthermore, two telecentres established by 
MSSRE had been closed and replaced by two alternative centres prior to the mission. The reasons for 
these closures should have been examined to bring out lessons for use in other centres. While the 
PANTLEG group wanted to focus on how telecentres can positively impact development, ignoring 
failures can lead to problems with program implementation and the sustainability of telecentres. Closely 
related to this experience is the concern that learning evaluations may be at risk of emphasising 

successes over failures because of inter- group competition, a desire to show hospitality, or incentives to 
gain funding or other support. An emphasis must be placed on honest sharing of information for the sake 
of learning. 


The PANTLEG group continues to develop its approach to evaluation (PANTLEG, 2000). Since their 
visit to India, PANTLEG has recognised five common causes of concern in telecentre operation and use 
including: standards for telecentre operation, policy environments for telecentre propagation, cultural 
context of telecentre deployment, sharing of knowledge between PANTLEG partners, and extending 
capabilities to non-PANTLEG individual and organisations. In order to address these points, 
PANTLEG has adopted comparative evaluation and exchange of participants, among other things, as 
crosscutting objectives for their work. A study mission to Sarawak has been proposed to evaluate the 
IDRC-supported remote community telecentre research project being carried out by the Universiti 
Malaysia Sarawak. A goal of the mission will be to further develop the evaluation methodology and to 
"define a pilot implementation of evaluation methodologies and techniques suitable for the Sarawak 
project and capable of enhancing the Group's capacity for evaluating telecentre activities" (PANTLEG, 
2000). They are also working towards developing adaptable and replicable learning evaluation methods 
that could eventually become part of a training program. 


PANTLEG in Light of the Guiding Principles for Telecentre Evaluation 
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Participatory - The PANTLEG approach raises a question as to the focus of the evaluation 
and choice of evaluator. Story collection was carried out by project leaders and researchers rather 
than telecentre operators and users. In order to be fully participatory the approach should empower 
telecentres, in addition to project co-ordinators, to carry out evaluations and include them in their 
planning processes. 


Socially Inclusive - PANLEG was aware of the need to be socially inclusive in the gathering of 
stories and made an effort to speak to various groups. By focusing on the individual, the approach 
provides a way to portray the experiences of various groups within a community - as long as all 
voices have an opportunity to be heard. 


Locally Grounded - This approach is very locally grounded in that it seeks the views of local 
telecentre operators and users on the impacts of the technology and the centre. The stories gathered 
will reflect the local experiences and attitudes towards the telecentre and uncover local distinctions 
and needs. 


Public and Transparent - PANLEG listened to stories in a public foram which makes the 
research process somewhat transparent. It is not clear, however, if the purpose, implications and 
results of the evaluation were made public to the study groups. 


Methodologically Appropriate - The applicability of this approach may depend on the 
situation. By its nature, it is flexible and easily adaptable however group facilitation and interviewing 
skills are required which may make it inappropriate for certain groups. It is a very practical 
approach in that stories are easily accessible and verifiable, however it is not a particularly robust 
methodology. There is a risk that survey groups will emphasise successes over failures in exchanges 
with other groups because of inter- group competition, a desire to show hospitality, or incentives to 
gain funding or other support. An emphasis needs to be placed on honest sharing for the sake of 
learning. Also, PANTLEG has explored the possibility of using story gathering throughout the 
project to complement base-line empirical data gathered at the beginning of the project. 


Sustainability Enhancing - This approach is very applicable to the promotion of quality and 
sustainability in that it provides an opportunity for learning exchanges and incremental improvement. 
However, this will require that telecentre operators, in addition to project co-ordinators, are trained 
on evaluation techniques. Also, telecentre operators and users would benefit from the exchange of 
experiences, successes and failures, solutions, and plans. 


Capacity Building - PANTLEG is still working on developing adaptable and replicable 
learning evaluation methods that could translate into a training program. Information exchanges build 
capacity both in terms of contributions to project and telecentre advances, and also in terms of 
evaluation capacity building. Serious reflection on the failures and successes of users and operators 
is empowering however, this goal would be further realised if telecentre operators were trained to 
carry out evaluations. PANTLEG should also put more emphasis on the often difficult task of 
understanding failures so that they can be avoided in the future. 


Reflective of Shared Visions - With reference to points 1 and 5, there remain questions 
around the interaction of telecentres, project co-ordination groups, and donors in the realisation of 
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the PANTLEG evaluations. It is not clear that telecentre user and operators have played a role in 
planning the evaluations. Also, stories are sometimes personal, so there needs to be some reflection 
on the ethical aspects of how the information is used. 


9. Strategically Oriented - Because this approach, in part, seeks to uncover unanticipated 
impacts, it could uncover information that challenges stakeholders. Storytellers may have their own 
agendas and thus the evaluator(s) cannot necessarily control the strategic orientation of the 
evaluation. An open-ended question is much more unwieldy than a set of survey questions, which 
can constrain the range of answers. Thus, this approach may be difficult to harness for strategic 
purposes. Stories can provide rich information on telecentre impacts, but may not be considered 
sufficiently robust by government agencies, donors or partners. For some purposes (policy 
development, telecentre management, economic implications), a different type of evaluation may be 
more appropriate. 


10. Gender Sensitive - It is important that women's voices are heard during the story collection 
process. When this is done, this approach allows women to express experiences without constraint. 
In turn stories can demonstrate how women are experiencing the telecentre. 


Experiences in Latin America: TELELAC - Exploring Evaluation Collectively 


IDRC's involvement with telecentre evaluation in the Latin American and Caribbean context has 
increasingly focused on its support of the Telecentre Evaluation Network in Latin America and the 
Caribbean Project also known as TELELAC (http://www. idrc.ca/pan/projects_e.htm#latinamerica) in 
addition to a small number of pilot telecentre implementation projects’. This project, which is being 
implemented by the Fundación ChasquiNet (http://www.chasquinet.org) of Ecuador, aims to support 
the development and strengthening of local, national and regional telecentres initiatives in Latin America 
and the Caribbean. The project's goals emerged out of a process that started with a survey and study 
conducted by IDRC staff followed by a discussion on the needs and expectations of telecentre 
operators and researchers on an electronic mailing list January - September, 1999). From this process 
it was determined that telecentre monitoring and evaluation is not strong in the region and in particular 
that: 


e Monitoring was often regarded as a project closure activity; 


e Monitoring during projects was mostly used as a means to correct errors or introduce efficiencies; 


e Evaluations have been either part of needs identification exercises, or carried out by external parties; 
e There was a need for a regional forum to communicating experiences, review methodologies, 
publicise results, and articulate regional and local dynamics. (Chasquinet, 2000b) 


Thus the major question faced by the TELELAC group was how to go about developing a collective 
evaluation process so that telecentres in the region could learn from each other through their 
experiences. The project's activities were elaborated first at the Far Hills meeting, and then during a 
planning meeting held in Papallacta, Ecuador in March, 2000 that included members of Chasquinet and 


3 Other telecentre projects in Latin America currently include Telecentres, Citizenship and Municipal Governance in Mexico, 
InforCauca Community Telecentres in Colombia, and Ashaninka - Internet by Radio for Indigenous Communities (Capacity 
Development for Internet use in Latin America and the Caribbean - LAC) in Peru. 
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other organisations which are actively involved in the project. The project is pursuing three main 
activities: 


1) research, analysis, diagnostics, documentation and creation of materials to be used by telecentre in 
Latin America and the Caribbean, as well as collection of historical telecentre experiences; 

2) the development of acollective space for hands on learning which will offer tools and services to 
strengthen telecentres; and 

3) an opportunity to develop, experiment and share a set of methods, techniques, and instruments for 
monitoring and evaluating the work of telecentres (Chasquinet, 2000a). 


Main goals of the project are to establish strategic alliances with selected beneficiary groups (especially 
marginalised groups), build user capacity, and train community facilitators. The philosophy of the project 
is based on democracy of access and knowledge, solidarity and mutual support, participation and 
transparency, proactive participation, and respect of diversity and gender equity. 


A set of guidelines was established at the Papallacta meeting to determine what level of participation 
would be necessary and acceptable for each component of the project (Chasquinet, 2000b). There is a 
co-ordination team consisting of Chasquinet members responsible for defining the structure and 
overseeing the implementation of the project. The 5-member assessment committee is made of up 
representatives of interested organisations and is responsible for ensuring that the goals of the project 
are met to the satisfaction of the members of TELELAC. The operational team consists of Chasquinet 
members as well as other interested parties and is responsible for the implementation of the project. All 
activities are to be carried out with reference to the TELELAC network through a project electronic 
mailing list and the existing TELELAC electronic mailing list. Each group or team also has a dedicated 
electronic mailing list to facilitate internal communication. 


The research component includes a quantitative inventory and qualitative analysis of telecentre 
initiatives in the region, and research to determine the "state of the art" in telecentre operations. Materials 
are being developed for use by telecentres with emphasis on collecting and documenting telecentre 
histories and experiences. The project is also stressing the importance of articulating local and regional 
dynamics given the close relationship between communities and telecentre operations. Two major 
research themes within this activity are to determine ways to adapt new technologies for communities 
with limited means, and to examine methods for diverse groups to influence telecommunications policy. 


The second main project activity is the development of an online telecentre resource clearinghouse 
(located at http://www.tele-centros.org) which will be used to strengthen collaborative work and 
exchange information. It will include a resource centre and toolkit with information and instructional 
materials relevant to telecentre operators and users. 


Finally the evaluation component will draw on the research component, as well as other investigations, 
to develop methodologies and techniques for monitoring and evaluating the work of telecentres. These 
materials will become part of the online toolkit. It is hoped that the monitoring and evaluation techniques 
will help operators and users to create a sustainable and meaningful telecentre institution. The 
methodology chosen for these activities is highly participatory focusing on: 


e learning through practice; 
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e using a multidisciplinary team for the development of tools involving, "grassroots communities, 
operators and promoters of telecentres working directly with grassroots groups, researchers and 
academics who are using these technologies,"; and 

e documenting and systematising the experiences of these groups to facilitate their dissemination and 
use. (ChasquiNet, 2000b) 


For the TELELAC group the development of monitoring and evaluation methodologies and techniques 
follows a transparent and participatory process of research and development. A major strategy is to 
focus on common priorities for monitoring and evaluation as determined through a collaborative and 
collective process of online discussions. The evaluation tools developed cover diagnostic tools, planning, 
management, and operations. The first step is to identify, analyse and adapt existing telecentre evaluation 
materials, with special attention to social, economic, cultural and gender parameters. This process is 
being complemented by the collection of existing telecentre evaluation results. An analytical framework 
and evaluation methodology will then be drafted, and revised after discussion with the TELELAC group. 


A 3-day workshop will be held to train a multi-disciplinary team of facilitators. These individuals will 
then undertake 3 pilot evaluations to test and refine the methodology, materials and instruments 
developed. A major goal of the pilot tests is to examine the adaptability and flexibility of the 
methodology to meet the specific needs of each pilot telecentre. Finally, a telecentre evaluation resource 
kit will be prepared and shared using the Tele-Centros site. This will include a monitoring system, an 
evaluation guide, and a training module for evaluation facilitators. The kit will also include a pool of 
evaluation experts, a continually updated set of lessons learned, bibliographies, and examples of 
methods and models used in various telecentres. 


The development of the project plan has been completed but according to Michel Menou (2000), one 
of the researchers participating in the project, work has yet to commence on the monitoring and 
evaluation methodologies. In analysing the TELELAC project, it is important to remember the following 
two points. 1) The TELELAC project is working on the development of a telecentre evaluation 
methodology, and not the evaluation of telecentres as such (except in the three pilot studies). And 2), 
The shape of the evaluation methodology to be developed cannot be predicted, especially since it is to 
be developed through a participatory process. On the other hand, the development process and 
philosophy is likely to be reflected in the final product. In Menou's view, the evaluation methodology to 
be developed should ensure that, 


The communities (groups of stakeholders) who are going to use/benefit from the telecentre 
should identify the expected social changes, specify how they can be œserved, and 
validate the observations. They should appoint representatives to a Monitoring and 
evaluation (M&E) committee which will assist the telecentre operators. Panels and group 
meetings should be implemented at regular (quarterly) intervals in order to discuss what is 
happening in relation to "key problems" illustrating the expected changes. External back up 
will be provided as appropriate and feasible, but most of the M&E should be carried out 
from within the stakeholders communities whose representatives should be trained for that 
purpose, together with telecentre operators. (Menou, 2000) 


Menou also envisions telecentre evaluation to focus on enhancing the effectiveness of the operation at 
the telecentre level; he sees telecentre evaluation as feeding into wider project evaluations. Finally, he 
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also points out that gender considerations will be a part of all phases of the project. According to 
Menou, "If designation of committee members or "key informants" does not secure fair representation of 
a particular group, or women, this would need to be corrected, possibly by duplicating relevant parts of 
the M&E with ad hoc panels. Whenever the special issues under review so require, women should be 
involved separately" (2000). 


This participatory approach can produce rich information and provides an opportunity for the people 
involved in the process to determine the research methodology. In this project, evaluation has meant 
trying to figure out where things are at; it is a process of "ubicación" (localisation). It is of particular note 
that this process is focused on the telecentre level and will allow telecentre users and operators to take 
ownership of developing and administering the evaluation process. Will the participatory forums 
undertaken by TELELAC translate into a viable approach to evaluation and locally and regionally 
applicable results? 


Another question surrounds the application of the methodology by telecentre users and operators. 
Based on the philosophy of the project and the attitudes of it researchers, we can predict that the 
resulting methodology will be convergent with the 10 Guidelines for Telecentre Evaluation. However, in 
the end, the strength of the approach will depend on how it is applied by telecentre users and operators. 
On the one hand, the existence of the Tele- Centros virtual community will help to ensure that this is done 
effectively at the telecentre level, and that experiences are shared at the regional level. But, the project 
will need to provide adequate training and documentation on the approach. In particular, since a goal of 
the methodology should be responsiveness to strategic orientations and local distinctions, information 
and training will need to be provided to advise the development of evaluation plans. The evaluation 
toolkit should also include complementary information to ensure its proper and effective application. This 
should cover participation, social inclusion, transparency, and gender. This information should ultimately 
contribute to an evaluation that is methodologically appropriate, reflective of shared visions, sustainability 
enhancing, and strategically oriented. 


Setting these concerns aside, the TELELAC approach has many merits. It is transparent and 
participatory, locally grounded, highly public, facilitates sustainability, and is supportive of capacity 
building efforts. This collective process will culminate in a collective result. By providing an opportunity 
for participation and capacity building at the telecentre and partner levels, the project is developing 
capacity for evaluation which will result in better local information and data for use in broader studies. 
Furthermore, this process of discovery will help the telecentre community to take ownership of the 
evaluation and research processes. The questions being asked are justifiable, but the results of the 
TELELAC project will be watched with interest. 


TELELAC in Light of the Guiding Principles for Telecentre Evaluation 


1. Participatory - The process for developing the TELELAC evaluation methodology is participatory 
and a criterion for the implementation of the project is that it be transparent and participatory. While 
it does not necessarily follow that the resulting methodology will be participatory, there is a strong 
indication - based on project philosophy, and on discussions with project members - that this will be 
a major consideration. 


2. Socially Inclusive - Until the actual methodology is developed it is difficult to judge whether it will 
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be socially inclusive, but once again, based on the project philosophy and interviews, it appears that 
this will be the case. It will also depend on how the methodology is applied by the individual 
telecentres, however the existence of the Tele-Centros network could help ensure that this goal is 
met. 


. Locally Grounded - TELELAC aims to develop an evaluation methodology that can be adapted 
to the context of the telecentre user. It will be challenging to develop a tool that will allow for 
aggregation or comparison of regional data, that is also flexible enough to accommodate local 
contexts, needs, visions and strategies. 


. Public and Transparent - The process for developing the evaluation methodology is highly public 
and transparent. All materials will be posted on the tele-centros website. However, once again, 
alignment with this criterion will be difficult to judge until the methodology is developed and will 
depend on independent applications. The Tele-Centros website will provide a means for telecentre 
groups to share their evaluation experiences and results. 


. Methodologically Appropriate - The development of a methodologically appropriate framework 
is the main goal of the evaluation portion of the TELELAC project. A major challenge will be to 
create a methodology that is flexible and adaptable to the specific needs of each telecentre, yet still 
provides a basis for regional comparisons. 


Sustainability Enhancing - It is the intention of the TELELAC project to enhance the sustainability 
of telecentres. This will be done by developing evaluation capacity through the establishment of 
appropriate methodologies and training, and fostering learning through the Tele-Centros virtual 
space. 


Capacity Building - Not only is the TELELAC project a capacity building exercise as such, but it 
will result in a set of resources and training exercises that will build that capacity of the participants in 
the project. Furthermore, the tele-centros virtual space will provide resources that will allow tele- 
centre practitioners and users to develop their capacity to more effectively use and operate 
telecentres. It will also provide a space for telecentre operators and users to exchange lessons 
learned. 


. Reflective of Shared Visions - The development of the TELELAC evaluation methodology is 
highly reflective of shared visions based on the results of surveys and group discussions. According 
to Menou's views, the methodology to be established should also support this type of process, 
however, once again this is yet to be determined. 


Strategically Oriented - The methodology developed must be flexible enough to be applied 
towards various ends. It will need to provide nformation for users and the community, for 
management, for outside agencies etc. Strategic orientation will depend on how the methodology 
gets applied and this should be taken into consideration in presentations of, or training on the 
material. The pros and cons of the methodology should be highlighted and situation-based 
alternatives should be provided. 


10. Gender Sensitive - Gender issues are being taken into consideration by the project team. Once 
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again, the gender sensitivity of the evaluation technique to be developed is yet to be seen, but 
current indications are that the criteria will be incorporated. It may be necessary to provide gender 
sensitivity training in order to ensure that the criterion is properly applied at the telecentre level. 


Conclusions 


By comparing the approaches to evaluation being developed by IDRC projects in, Asia and Latin 
America against the Guidelines for Telecentre Evaluation we can see several regional and institutional 
differences. PANTLEG is highly focused on the individual and on people, making the approach 
participatory, socially inclusive, locally grounded and sensitive to gender concerns. TELELAC has 
focused on the need to develop collective processes for fostering capacity and sustainability, and on 
dissemination strategies. 


There are common threads that can be drawn from the experiences. Reflecting on the discussion of 
evaluation purposes presented in the introduction, this paper constitutes evaluation of the evaluation 
system, or meta-evaluation. Meta-evaluation can be a part of the ongoing process of the regional 
experiences as they work to develop their approach, methodologies and tools. In both cases, the 
development of methodologies and tools is an ongoing process with PANTLEG working on story-telling 
techniques, TELELAC focusing on collective discussions of appropriate tools. Training goes hand in 
hand with the development of an approach and again, this was an issue in both cases. PANTLEG will 
need to provide facilitation training for telecentre operators, and TELELAC should prove training 
materials to accompany its online tool kit in order to ensure that tools are properly implemented. Finally, 
both approaches share a need to disseminate results. In the case of TELELAC, dissemination channels 
have been established, but in the other case, strategies are weak in this area. 


Both approaches have provided an opportunity to reflect on the purpose of evaluations in the context of 
telecentres and the role of donor agencies in facilitating them. The figure below compares the main focus 
of the evaluation strategy in each of the cases. PANTLEG has focused on collecting data from individual 
telecentres, with participation of both program officers and telecentre users and staff. However, to date, 
recipients and IDRC staff have used this information, not individual telecentres. TELELAC is working 
on developing a methodology and communication strategy to facilitate evaluations at the local level, 
primarily for the benefit of telecentre operators and users. However, the project’s online data gathering 
and communications strategies will provide a means to gather information of use for bigger research 


questions. 
Focus of the Evaluation Asia Latin America 
Strategy 


The Big Questions 


Local Needs TELELAC 


Evaluations have both academic and more practical drivers and this is particularly true of the new field of 
telecentre evaluation. Donors have a responsibility not only to support the useful and financially 
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responsible realisation of telecentre projects, but they also have a role to play in analysing and theorising 
on the impact of telecentres. We need to recognise that this is an inherently political task. As Menou 
points out, "Policy and decision makers who are approving or supporting programs for the development 
of the Internet, would like to have some sort of evidence in support of the claims by the proponents of 
themselves that it will bring about all the said benefits" (Menou, 1999). While research on telecentres is 
being done by way of evaluation, it is important to remember that the methodologies we use play a role 
in the research results. People have a right to have a voice in the results of telecentre evaluations 
because the research generated will determine the future direction of telecentre activities. 


But at the same time, it is important to remember that individual telecentres have specialized needs and 
will require training and support to carry out local, specialized evaluations. Local research programs are 
also important for the identification of best practices and impacts. Their findings should be incorporated 
into the objectives of programs in the region to be reflected in future program evaluations. To facilitate 
this type of research, IDRC and its partners need to develop guidelines or techniques for local small- 
scale evaluation that are easy to implement at a low cost, and champion their application in telecentres 
through program work. Better local information can not only facilitate better project results, but can also 
provide a basis for better program evaluations and research. Finally, IDRC should contribute to the use 
of evaluation approaches that facilitate the participation of partners and telecentres in the research 
process so that they have a role in determining the parameters of their work. 


This analysis of IDRC telecentre evaluation experiences in Asia, and Latin America has demonstrated 
the usefulness of the Guidelines for Telecentre Evaluation as they provide a common framework for 
assessing each experience. The guidelines can be used to assess other telecentre experiences and allow 
us to determine if telecentre evaluation is being useful, financially responsible, if it is building local 
capacity and enabling lesson sharing. IDRC is also using the guidelines as a starting point as it works 
towards developing a centre- wide framework for ICT evaluation. 
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